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Abstract 

Background: The purpose of this article is to report on the quality of the existing evidence base regarding the 
effectiveness of clinical pathway (CPW) research in the hospital setting. The analysis is based on a recently 
published Cochrane review of the effectiveness of CPWs. 

Methods: An integral component of the review process was a rigorous appraisal of the methodological quality of 
published CPW evaluations. This allowed the identification of strengths and limitations of the evidence base for 
CPW effectiveness. We followed the validated Cochrane Effective Practice and Organisation of Care Group (EPOC) 
criteria for randomized and non-randomized clinical pathway evaluations. In addition, we tested the hypotheses 
that simple pre-post studies tend to overestimate CPW effects reported. 

Results: Out of the 260 primary studies meeting CPW content criteria, only 27 studies met the EPOC study design 
criteria, with the majority of CPW studies (more than 70 %) excluded from the review on the basis that they were 
simple pre-post evaluations, mostly comparing two or more annual patient cohorts. Methodologically poor study 
designs are often used to evaluate CPWs and this compromises the quality of the existing evidence base. 

Conclusions: Cochrane EPOC methodological criteria, including the selection of rigorous study designs along with 
detailed descriptions of CPW development and implementation processes, are recommended for quantitative 
evaluations to improve the evidence base for the use of CPWs in hospitals. 



Background 

What are clinical pathways? 

Clinical pathways (CPWs) are evidence-based multidis- 
ciplinary care plans which describe the essential steps 
needed in the care of patients with a specific clinical 
problem. They are used to translate clinical guidelines 
into local protocols and clinical practice [1]. Whereas 
clinical guidelines provide generic recommendations, 
CPWs are specifically tailored to the local hospital struc- 
tures, systems and time-frames used. 

Clinical pathways have been proposed as a strategy to 
optimise resource allocation in a climate of increasing 
healthcare costs [2]. Other terms used to describe clinical 
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pathways include 'integrated care pathways, ' 'critical path- 
ways, ' 'care plans, ' 'care paths' and 'care maps.' 

Objectives 

The first objective of this article is to report on the 
methodological quality of the existing evidence base 
regarding the effectiveness of CPW research in the hos- 
pital setting. An international, multidisciplinary team of 
researchers conducted a systematic review of the effect- 
iveness of CPWs in hospitals, with the findings recently 
published in the Cochrane library [3]. The second ob- 
jective is to test the hypothesis that simple pre-post 
studies tend to overestimate CPW effects reported. 

Method 

We followed the validated Cochrane Effective Practice 
and Organisation of Care Group (EPOC) methodology 
for considering and analysing studies [4]. The primary 
systematic review aimed to catalogue the international 
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evidence to assess the effect of clinical pathways on pro- 
fessional practice, patient outcomes, length of hospital 
stay and hospital costs. We searched the Database of 
Abstracts of Reviews of Effectiveness, the Effective Prac- 
tice and Organisation of Care Register, the Cochrane 
Central Register of Controlled Trials and bibliographic 
databases including MEDLINE, EMBASE, CINAHL, 
NHS EED and Global Health. Details of the electronic 
search strategy for the identification of studies are pre- 
sented in detail in the EPOC review, recently published 
in the Cochrane Library [3]. Our team developed and 
validated five minimum criteria to define a CPW to en- 
sure that only appropriate studies were sourced and 
included in the review [5]. An integral component of the 
review process was a rigorous appraisal of the study 
designs and methodological quality of all relevant CPW 
evaluations. This allowed the identification of strengths 
and limitations of the evidence base for CPW effective- 
ness with regard to the first study objective. 

Assessment of study design 

For the purpose of the systematic review on CPWs in 
hospitals, four study designs were considered for inclu- 
sion: randomized controlled trials (RCTs), controlled 
clinical trials (CCTs), controlled before and after studies 
(CBAs) and interrupted time series analysis (ITS). 

While there are many well developed and well 
accepted critical appraisal criteria for experimental stud- 
ies, fewer exist for non-experimental studies such as 
CBAs and ITS. Both designs are subject to a lack of con- 
trol and high risk of bias so EPOC developed criteria to 
facilitate their quality assessment and inclusion (where 
appropriate) in systematic reviews. For example, CBAs 
are required to have more than one control group and 
ITS require at least three time points before and after an 
intervention. Validated criteria for the assessment of 
these designs have been developed by EPOC and are 
available from the EPOC website [6] and the four differ- 
ent study designs are briefly outlined in Table 1. In 
addition, the simplified EPOC gold standard of study 
designs considered for inclusion in the present review 
are depicted in Figure 1 [4] . 

Risk of bias assessment 

We developed a quality assessment and data abstraction 
instrument incorporating the EPOC risk of bias criteria 
[4]. Quality assessment was conducted on full-text arti- 
cles once initial literature searching and screening indi- 
cated that articles were research-based and referred to a 
CPW and were, subsequently, potentially relevant. The 
EPOC approach for judging risk of bias of randomized 
and non-randomized studies is a two-part assessment 
tool, concerning specific domains and quality criteria 
(i.e. RCTs: sequence generation, allocation concealment. 



blinding, et cetera). The validated risk of bias criteria can 
be found in the Cochrane EPOC Group module [4] and 
are presented in detail in additional file 1. 

Comparison of CPW interventions 

We compared patients managed according to CPW to 
those managed by usual care, and patients treated within 
a multifaceted intervention including a CPW compared 
to usual care. 

Secondary analysis 

The aim of the secondary analysis was to determine 
whether pre-post study design was associated with an 
overestimate of the effects of CPW. Other researchers 
also compared the findings of randomized evaluations 
vs. non-randomized study designs and concluded that 
such studies potentially overestimate the effects reported 
and there were systematic differences between effects 
estimated [7-9]. To test the hypothesis, we compared 14 
primary studies [10-23], included in the Cochrane re- 
view, grouped into category 1 (patients managed accord- 
ing to CPW compared to usual care), and reporting on 
length of stay (LOS) as the most commonly employed 
outcome measure with a randomly selected sample of 14 
excluded pre-post CPW evaluations also reporting LOS 
[24-37]. The selection of a random sample of studies 
was taken from those studies excluded on the basis of a 
simple pre-post design not meeting EPOC quality criteria 
(see Table 2). We used a computer generated random 
sample (RAND function in Excel) [38] of 14 excluded pre- 
post studies reporting LOS as a primary study outcome 
[24-37]. 

Statistical pooling (meta-analysis) 

A Cochrane web-based program. Review Manager 
(RevMan), was used to calculate a pooled estimate of 
the combined intervention effect on LOS, called 
weighted mean difference (WMD) [39]. We used a ran- 
dom effects model since this model estimates the effect 
with consideration to the variance between studies, ra- 
ther than ignoring heterogeneity by employing a fixed 
effect model [40]. Statistical inconsistency within both 
subgroups was assessed by calculating a test of hetero- 
geneity (I square (I^)). 

Results 

All potentially relevant studies were assessed using the 
CPW definition [5] and EPOC review inclusion criteria 
for acceptable study designs [4]. Using two independent 
reviewers, we rejected 2954 of the 3214 potential papers 
and only 260 primary studies were initially identified as 
potentially relevant and full text copies were retrieved. 
Figure 2 illustrates the described trial flow. 
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Table 1 EPOC study designs considered for inclusion 



Patient randomized 
controlled trials 
(P-RCT): 

Cluster randomized 
controlled trials 
(C-RCO: 



Non-randomized 
controlled trials 
(CCTs): 

Controlled before 
and after studies 
(CBAs): 



The individual patients are allocated by random to the intervention or control group. Individual randomisation 
facilitates equally distributed patient characteristics and comparability. Only the exposure to the intervention should 
be the factor that distinguishes between both groups. 

This is a robust study design that prevents contamination of professionals by randomising groups of professionals 
(i.e. different practices, wards or hospitals). However, this means the fundamental assumption of independence 
is violated because patients within a cluster are more likely to respond in a similar manner. This lack of 
independence, statistically called "intracluster correlation," also means a specific adjustment for clustering effects is 
required to assure comparability with individually randomized trials. 

Patient or cluster trials where allocation to experimental and control groups is quasi-random (i.e. alternated allocation). 



CBAs are experimental studies with two or more control groups compared with one or more experimental groups 
but allocation is not random. Data is collected on the control and intervention groups before the intervention is 
introduced and then further data is collected after the intervention has been introduced. The reliability of the 
intervention effect is questionable because there may be unidentified differences between the experimental intervention 
and control groups which may have modified the observed effect. Note: EPOC has recently changed the policy about 
inclusion of CBA studies with only one intervention site. Specific details about design criteria can be found at the 
website (www.epoccochrane.org) 

nterrupted time This represents a robust method of measuring the effect of an intervention as a trend over time. It is a useful design 

series designs when recruitment of a control cohort is impractical, e.g. due to changes in hospital policy. Three or more data points 

(ITS): are collected before and after the intervention as a minimum standard. The intervention effect is measured against 

the pre-intervention trend. 

Source: Bero L, Eccles M, Grimshaw J, Gruen RL, Mayhew A, Oxman AD, lavender E, Zwarenstein M, Shepperd S, Paulsen E, Pantoja T, Lewin S, Ballini L. Cochrane 
Effective Practice and Organisation of Care Group (Cochrane Group IVlodule). About The Cochrane Collaboration (Cochrane Review Groups (CRGs)). The Cochrane 
Library. Oxford: John Wiley, 2009; adopted by the authors. 



The majority (79.1 %) of the rejected studies (2335 out 
of 2954) had to be excluded because they failed to meet 
our definition of CPW. Table 2 illustrates the reasons for 
exclusion following title and abstract review. Out of the 
260 primary studies meeting CPW content criteria fol- 
lowing review of the full text, only 27 studies met the 
EPOC study design and risk of bias criteria. 

Assessment of study design 

Out of 27 CPW evaluations included, nineteen of the 
included studies were randomised controlled trials 
(RCTs) [10,11,13-22,41-48], including two cluster rando- 
mised trials (C-RCT) [20,47]. Four studies were CBAs 
[49-52], two were CCTs [12,23] and two ITS [53,54]. 

Of the original studies which met the CPW content 
criteria, more than 70 % were excluded from the review 
as they were simple pre-post evaluations, mostly com- 
paring two or more yearly patient cohorts (see Table 3). 

Risk of bias assessment 

Out of the 228 studies excluded in phase two following 
full text review (see Table 3) only four non-randomized 
studies [55-58] and one randomised clinical study (RCT) 
[59] were excluded because of high risk of bias. The 
RCT from Bittinger (1995) did not meet EPOC quality 
criteria as only 50 % of study patients were followed up 
after randomization and there was a high risk of attrition 
bias. Four time series studies were excluded as data was 
not analyzed appropriately. The studies from Joiner 



(1996), Smith (1999), Summers (1998) and Warner 
(2002) had a high risk of bias because no statistical con- 
trol was used [55-58]. 

Table 3 illustrates the reasons for exclusion in stage 
two after meeting CPW content criteria in stage one. 

Secondary analysis 

In Figure 3 we provide the detailed results of the meth- 
odological comparison of the 14 included primary studies 
which utilised Cochrane EPOC study design quality cri- 
teria [4] and reporting on LOS [10-23] vs. 14 randomly 
selected pre-post studies excluded from the review and 
reporting on LOS as a primary outcome [24-37]. We 
observed considerable statistical inconsistency within 
both subgroups of CPW studies, so the calculated esti- 
mates in LOS per subgroup should be treated with caution 
(1^ = 62% Cochrane EPOC subgroup vs. 98% randomly 
selected subgroup.) 

We observed greater reported LOS effects within the 
random subgroup of excluded pre-post studies after meet- 
ing CPW content criteria (WMD - 2.85 (95%CI: - 3.58 to 
- 2.11)), versus the pooled LOS data recently published 
in the Cochrane library (group 1 clinical pathway vs. 
usual care WMD - 1.23 (95%CI - 1.73 to - 0.72)) [3]. 
Moreover, the pre-post studies in the randomly selected 
subgroup tend to report more consistently on signifi- 
cant reductions in LOS (see Figure 3). Statistically, the 
chi-squared test for subgroup differences also reached a 
significant level (P = 0.0004). 
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Randomised controlled trial (RCT) 

2 or more groups, random allocation 
Possible participants randomly 
allocated to either 



Inlen/ention group | ( Control group 



Controlled clinical trial (CCT) 

As for a RCT but the allocation was quasi-random (eg. alternation, date of birth, patient 

identifier) 



Possible participants guasi-randomlv 
allocated to either 



Intervention group 



Control group 



Controlled before and after study (CBA) 

3 or more groups but allocation to control or intervention was not random, and there are 
at least 2 control groups 

Possible participants 
non-randomlv allocated 
to either 



Inten/ention group - 
one or more than one 
group 



Control group - must 
be more than one 
group 



Control group - must 
be more than one 
group 



Note: EPOC has recently ctianged the policy about inclusioh ot CBA studies with only one inten/ention site. Specitic 
details about design criteha can be found at the website (www.epoc.cochrane.org) 



Interrupted time series (ITS) 

For assessing a trend over time. No control group but must have a clear point In time 
that intervention was introduced, plus 3 data points before and after the Intervention. 

Time 1 

Time 2 

Time 3 
Intervention introduced 

Time 4 

Time 5 

Times 

If none of these designs then paper is EXCLUDED . 



Figure 1 Simplified EPOC standard of study designs considered for inclusion in the present review. Source: Bero L, Eccles M, Grimshaw J, 
Gruen RL, Mayhew A, Oxman AD, Tavender E, Zwarenstein M, Shepperd S, Paulsen E, Pantoja T, Lewin S, Ballini L. Cochrane Effective Practice and 
Organisation of Care Group (Cochrane Group Module). About The Cochrane Collaboration (Cochrane Review Groups (CRGs)). The Cochrane 
Library. Oxford: John Wiley, 2009; adopted by the authors. 



Discussion 

Why is it important to critically appraise study designs In 
a systematic review? 

We followed the validated Cochrane EPOC criteria for 
randomized and non-randomized CPW evaluations [4,6]. 
The finding that the vast majority of studies failed to 
meet methodological quality criteria strongly indicates 

Table 2 Reasons for exclusion stage one (n = 2954) 



Reason 


Number 


% 


Not CPW 


2335 


79.1 


Not study 


253 


8.6 


Not hospital 


246 


8.3 


EPOC minimum study design criteria not met 


89 


3.0 


Other (e.g. qualitative study) 


31 


1.0 


Total 


2954 


100 



that low quality study designs are too often used to 
evaluate CPWs and contribute very little to the evidence 
base regarding CPWs. 

Many of such excluded CPW evaluations claimed to pro- 
vide evidence for the effectiveness of the pathway interven- 
tion under consideration but, with a methodologically weak 
study design, it remains unclear if the reported effect was 
really attributable to the CPW effectiveness or any other 
unknown factors. Possible confounding factors might have 
been the case-mix introduction, hospital quality improve- 
ment initiatives or changes in hospital policy [2]. The un- 
controlled nature and exposure to bias convey that such 
studies contribute very little to the evidence-base. 

Implications of including weak study designs 

Based on our review experience, we reaffirm that uncon- 
trolled pre-post designs are commonly used to evaluate 
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Stage One: 
Abstracts screened 
n=3214 



Included 
n = 260 



Excluded 
n = 2954 



Stage Two: 

Full-text quality 
assessment 



Included Excluded Pending 



Figure 2 Trial flow. 



the effectiveness of CPWs. Such designs are likely to be 
misleading and contribute little to understanding the 
reported effects of pathways. Considering the second ob- 
jective of this article, the meta-analytic comparison sup- 
ports other evidence [7-9] that simple pre-post study 
designs tend to overestimate intervention effects reported. 

There is a place for well designed process-evaluations 
also referred to as interrupted time series (ITS) to ex- 
plore and provide more insights into the varying path- 
way components and their causal effectiveness to 
determine how CPW interventions actually work. Care- 
fully designed time series studies are less resource- 
intensive than RCTs, do not require a control group, 



Table 3 Reasons for exclusion following full text review 
(n = 228) 



Reason 


Number 


% 


Not CPW 


38 


16.7 


Simple preVpost evaluations 


160 


70.2 


High risk of bias 


5 (IRCT) 


2.2 


Not study 


14 


6.1 


Not hospital 


11 


4.8 


Total 


228 


100 



and allow for the use of retrospective data. While requir- 
ing more advanced statistical techniques than simple 
pre-post studies, ITS supports research outcomes that 
are more likely to contribute to the evidence base, in- 
cluding systematic reviews. Better designed, conducted 
and reported CPW evaluations will contribute to a bet- 
ter understanding of the key elements of CPWs that im- 
pact on patient, provider and economic outcomes. 

Limitations 

The majority of included studies employed LOS as a 
performance measure. Hence, we compared the magni- 
tudes of CPW effects on length of stay (n = 14 primary 
studies) rather than patient outcomes such as mortality 
(n = 4 studies) or in-hospital complications (n = 5 studies) 
[3]. The low number of primary CPW evaluations 
included in the review which reported on patient out- 
comes prevented further testing of the robustness of this 
methodological comparison. 

Conclusion 

Cochrane EPOC methodological inclusion criteria should 
be considered for quantitative evaluations into the impact 
of CPWs in hospitals. Based on our review experience, the 
EPOC methodological gold standard is infrequently 
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Joh 2003 
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Kelly 2000 


1.51 


0.58 


34 
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42.8 


35 


37 


42.8 


35 


0.1% 


-20.40 (-40.45, 


-0.35] 


4- 


Ogawa 2004 


12 


1 


46 


22 


3.2 


26 


7.7% 


-10.00 (-11.26, 


-874] 


< 


Perlstein 2000 


2.4 


1.3 


679 


2.9 


2 


1300 


10.0% 


-0.50 (-0.65, 


-0.35] 




Pestian 1998 


0.6 


0.3 


40 


0.7 


0.2 


40 


10.0% 


-0.10 (-0.21 


0.01] 




Turley 1994 


3.72 


1.01 


58 


662 


2.65 


58 


9.1% 


-2.90 (-3.63, 


-2.17] 




Walsh 2001 


8.2 


5.3 


58 


10.3 


5.7 


67 


5.9% 


-2.10 (-4.03, 


-0.17] 




Wilson 2002 


2.09 


1.3 


96 


2.55 


1.3 


85 


9.8% 


-0.46 (-0.84, 


-0.08] 




Yamauchi 2003 


10.9 


2 


17 


16.2 


6.9 


96 


66% 


-5.30 (-6.98, 


-3.62] 




Slll)tot<Tl (95»o CD 






1942 






2511 


100.0«o 


-2.85 [-3.58. 


-2.11] 




Heterogeneity: Tau== 1 


42; Chi= 


= 542. 


22, df = 


13 (P < 


0.00001); r = 


98% 









Test for overall effect: Z = 7.56 (P < 0.00001) 



Test for subgroup differences: Chi°= 12.55 df= 1 (P = 0.0004). I' = 92.0% 



-4 -2 0 2 4 
Favours pathway Favours usual care 



Figure 3 Comparison meta analysis LOS COCHRANE vs. Excluded pre-post studies. 



transferred into research practice. Future evaluators could 
hereby contribute significantly to the understanding of 
factors associated with the reported effects of clinical 
pathways in hospitals by incorporating EPOC criteria into 
study design. Whilst experimental methods such as rando- 
mised trials are recommended they may be considered be- 
yond the capacity of many clinicians and researchers. A 
well designed evaluation such as ITS or CBA that meets 
the EPOC gold standard methodological criteria can pro- 
duce meaningful, rigorous results with the use of rela- 
tively few resources. In terms of the second study 
objective, the methodological comparison of Cochrane 
vs. non Cochrane study designs (see Figure 3) also sup- 
port the finding that simple pre-post study designs tend 
to overestimate CPW effects reported. 
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Additional file 1: EPOC risk of bias criteria. 
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